[Dart-dev] [7853] DART/trunk/models/cam/shell_scripts: The major upgrades and changes include the following.
nancy at ucar.edu
nancy at ucar.edu
Tue Apr 7 15:52:47 MDT 2015
Revision: 7853
Author: thoar
Date: 2015-04-07 15:52:47 -0600 (Tue, 07 Apr 2015)
Log Message:
-----------
The major upgrades and changes include the following.
More detailed comments are interspersed in the log since late 2013.
1) Addition of WACCM to the list of CAM variants which can be used in assimilations.
This required the addition of a new vertical coordinate; scale height ('log_invP'),
which has a small but widespread footprint.
2) Addition of CAM-SE to the list of CAM variants.
This required new interpolation routines and grid file management to handle
the logically non-rectangular grid (cubed sphere).
This consists mostly of a large new section of the module, but some scattered
changes as well.
3) Fixing the misinterpretation of DART KINDs vs TYPEs.
This was fixed independently on the trunk.
4) Code review leading to some structural and extensive cosmetic changes to bring
it into line with DART practices.
5) Rewriting of get_close_obs to handle the DART QC status better and
improve the clarity and logic, as well as accomodate items 1)-4).
6) Implementation of an improved damping algorithm for the top layers of the state,
where CAM applies enhanced diffusion and the assimilation should be weaker.
This algorithm has the right units, works for vert_coord = 'pressure' and
'log_invP', and allows more of the state to feel the observations.
7) Inclusion of scripting which will enable running multiple assimilation cycles
in a single job, which eliminates much of the waiting in the queue, and saves
~30% in computing time by running st_archive.sh offline as a single processor job.
This is accomplished through the creation of scripts ${CASE}.submit_cycles and
${CASE}.run_cycles by the new CESM1_2_1_setup_advanced. CESM1_2_1_setup_hybrid
now exists as the streamlined, 'beginners' script, which trades some of the
capabilities in CESM1_2_1_setup_advanced in favor of ease of use.
The CAM-FV version from branches/cam was tested against the trunk.
When the improvement in the choice of PS location in convert_vert is taken
into account, the results are identical.
No direct comparisons against the trunk are possible for WACCM or CAM-SE.
Those were compared qualitatively against the branches' CAM-FV assimilations.
A notable limitation discovered during testing is that filter can fail
for task counts > ~4200, due to an MPI imbalance. Filter completes all of
its work, but never returns from MPI_finalize. Work-arounds are available,
but are not included in the trunk.
At this time the scripts may only work for setting up the CAM5 version of WACCM.
The CAM4 version forecasts fail, probably due to input/forcing file incompatibility.
This is at the top of the 'fix-it' list.
These scripts do not support variable-resolution grids.
If you need them, they are available in branches/cam.
Modified Paths:
--------------
DART/trunk/models/cam/shell_scripts/CESM1_2_1_setup_hybrid
DART/trunk/models/cam/shell_scripts/CESM1_2_1_setup_pmo
DART/trunk/models/cam/shell_scripts/archive_cycles.csh
Added Paths:
-----------
DART/trunk/models/cam/shell_scripts/CESM1_2_1_setup_advanced
DART/trunk/models/cam/shell_scripts/CESM1_2_1_setup_pmo_advanced
-------------- next part --------------
Added: DART/trunk/models/cam/shell_scripts/CESM1_2_1_setup_advanced
===================================================================
--- DART/trunk/models/cam/shell_scripts/CESM1_2_1_setup_advanced (rev 0)
+++ DART/trunk/models/cam/shell_scripts/CESM1_2_1_setup_advanced 2015-04-07 21:52:47 UTC (rev 7853)
@@ -0,0 +1,1205 @@
+#!/bin/csh -f
+#
+# DART software - Copyright 2004 - 2013 UCAR. This open source software is
+# provided by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+#
+# DART $Id: CESM1_2_1_setup_advanced 7634 2015-02-26 21:56:52Z raeder $
+
+#*******************************************************************************
+#
+# ---------------------
+# Purpose
+# ---------------------
+#
+# This script is designed to set up, stage, and build a multi-instance run
+# of CESM using an F compset where CAM, CLM and CICE are active. The initial state
+# can come from a single multi-instance reference case so a CESM hybrid setup
+# is used. Instructions on what to change to use the SE core or WACCM are
+# outlined in the models/cam/model_mod.html documentation.
+#
+# This script differs from the "simple" CESM1_2_1_setup_pmo in several ways.
+# Multiple cycles may be completed in a single job, a series of
+# dependent jobs may be queued more-or-less automatically, and the archiving
+# is explicitly performed by a separate single-threaded job. This is a substantially
+# more efficient use of resources than having a large-CPU job perform an inherently
+# single-threaded task. This requires a large run directory and knowledge of how
+# many cycles may be completed before the archiving must happen.
+# This script also has examples of how to specify alternate support filles that
+# cover more recent dates than the current defaults. For more detail, consult
+# the models/cam/shell_scripts/README_advanced and the models/cam/model_mod.html.
+#
+# 'Cycles' marks changes to set up scripts which can run multiple forecast+assim cycles
+# in one job, and use a much more efficient, offline, short term archiver.
+# Run the first cycle in the usual way, since it needs to be CONTINUE_RUN = FALSE.
+# Then switch to the cycles scripts in $CASEROOT:
+# -> ${case}.submit_cycles (edit this to determine number of cycles/job, etc.)
+# -> submits $case.run_cycles
+# -> calls the next $case.run_cycles, NOT as a separate job.
+# -> archive_cycles.csh (runs when the right number of *jobs* have finished)
+# archives the desired restart files
+# archives the history files
+# -> lt_archive.sh -m copy_dirs_hsi
+#
+# Variable resolution in CAM-SE is not yet available in the public releases of CESM.
+#
+# DOCN: We are using a single data ocean.
+#
+# Because the atmosphere assimilations typically occur every 6 hours,
+# the methodology here reflects that. All of CESM stops every 6 hours
+# so that a CAM output file would be available for assimilation.
+#
+# CESM/DART requires some modifications to the CESM source code EVEN IF YOU
+# ARE NOT USING DART. The modifications for CAM require a change to the CESM
+# ${CASE}.run script to invoke a DART script that will allow CAM to cycle
+# correctly with the source code modifications. Changing one line will
+# allow you to invoke DART.
+#
+# This script results in a viable setup for a CESM multi-instance experiment.
+# You are STRONGLY encouraged to run the multi-instance CESM a few times and
+# experiment with different settings BEFORE you try to assimilate observations.
+# The data volume is quite large and you should become comfortable using
+# CESM's restart capability to re-stage files in your RUN directory.
+#
+# ${CASEROOT}/CESM_DART_config is automatically run by this script and will
+# augment the CESM case with the required setup and configuration to use DART
+# to perform an assimilation. CESM_DART_config will insert a few dozen
+# lines into the ${case}.run script after it makes a backup copy.
+#
+# This script relies heavily on the information in:
+# http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/usersguide1_2/book1.html
+#
+# ---------------------
+# How to use this script.
+# ---------------------
+#
+# -- You will have to read and understand the script in its entirety.
+# You will have to modify things outside this script.
+# This script sets up a plain CESM multi-instance run without DART,
+# intentionally. Once it is running, calls to DART can be added.
+#
+# -- Examine the whole script to identify things to change for your experiments.
+#
+# -- Edit this script in the $DART/models/cam/shell_scripts directory
+# or copy it to somewhere where it will be preserved.
+#
+# -- Locate the initial multi-instance files that CESM will need.
+#
+# -- Run this script. When it is executed, it will create:
+# 1) a CESM 'CASE' directory, where the model will be built,
+# 2) a run directory, where each forecast (and assimilation) will take place,
+# 3) a bld directory for the executables.
+# 4) The short term archiver will use a fourth directory for
+# storage of model output until it can be moved to long term storage (HPSS)
+#
+# This script also executes ${CASEROOT}/CESM_DART_config to modify
+# the ${CASEROOT}/${CASE}.run script so that the SourceMods for CAM
+# are effective. CESM_DART_config will also augment the case with all
+# the pieces necessary to run DART when the time comes.
+#
+# -- (if running DART) Edit the DART input.nml that appears in the ${CASEROOT}
+# directory.
+#
+# -- Submit the job using ${CASEROOT}/${CASE}.submit
+#
+# ---------------------
+# Important features
+# ---------------------
+#
+# If you want to change something in your case other than the runtime
+# settings, it is safest to delete everything and start the run from scratch.
+# For the brave, read
+#
+# http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/usersguide1_2/x1080.html
+#
+# and you may be able to salvage something with
+# ./cesm_setup -clean
+# ./cesm_setup
+# ./${case}.clean_build
+# ./${case}.build
+#
+#*******************************************************************************
+
+# ==============================================================================
+# case options:
+#
+# case The value of "case" will be used many ways; directory and file
+# names both locally and on HPSS, and script names; so consider
+# its length and information content.
+# compset Defines the vertical resolution and physics packages to be used.
+# Must be a standard CESM compset; see the CESM documentation.
+# resolution Defines the horizontal resolution and dynamics; see CESM docs.
+# T85 ... eulerian at ~ 1 degree
+# ne30np4_gx1v6 ... SE core at ~ 1 degree
+# f09_f09 ... FV core at ~ 1 degree
+# BUG 1384 may apply, check if ocean and atm/land must be at same resolution.
+# cesmtag The version of the CESM source code to use when building the code.
+# A directory with this name must exist in your home directory,
+# and have SourceMods in it. See the SourceMods section.
+# http://www.image.ucar.edu/pub/DART/CESM/README
+# num_instances The number of ensemble members.
+#
+# Guidelines on what to change for an SE or WACCM run are described in the
+# models/cam/model_mod.html documentation.
+# ==============================================================================
+# AMIP_CAM5_CLM40%SP_CICE%PRES_DOCN%DOM_RTM_SGLC_SWAV (F_AMIP_CAM5) (FAMIPC5)
+
+setenv case cam1_2_1_hybrid
+setenv compset F_AMIP_CAM5
+setenv resolution f09_f09
+setenv cesmtag cesm1_2_1
+setenv num_instances 30
+
+# ==============================================================================
+# machines and directories:
+#
+# mach Computer name
+# cesmdata Location of some supporting CESM data files.
+# cesmroot Location of the CESM code base. This version of the script
+# only supports version cesm1_2_1.
+# caseroot Will create the CESM case directory here, where the CESM+DART
+# configuration files will be stored. This should probably not
+# be in scratch (on yellowstone, your 'work' partition is suggested).
+# This script will delete any existing caseroot, so this script,
+# and other useful things should be kept elsewhere.
+# rundir Will create the CESM run directory here. Will need large
+# amounts of disk space, generally on a scratch partition.
+# exeroot Will create the CESM executable directory here, where the
+# CESM executables will be built. Medium amount of space
+# needed, generally on a scratch partition.
+# archdir Will create the CESM short-term archive directories here.
+# Large, generally on a scratch partition. Files will remain
+# here until the long-term archiver moves it to permanent storage.
+# dartroot Location of the root of _your_ DART installation
+# ==============================================================================
+
+setenv mach yellowstone
+setenv cesmdata /glade/p/cesm/cseg/inputdata
+setenv cesmroot /glade/p/cesm/cseg/collections/$cesmtag
+setenv caseroot /glade/p/work/${USER}/cases/${case}
+setenv rundir /glade/scratch/${USER}/${case}/run
+setenv exeroot /glade/scratch/${USER}/${case}/bld
+setenv archdir /glade/scratch/${USER}/archive/${case}
+setenv dartroot /glade/u/home/${USER}/svn/DART
+
+# ==============================================================================
+# configure settings:
+#
+# refcase The name of the existing reference case that this run will
+# start from.
+#
+# refyear The specific date/time-of-day in the reference case that this
+# refmon run will start from. (Also see 'runtime settings' below for
+# refday start_year, start_mon, start_day and start_tod.)
+# reftod
+# NOTE: all the ref* variables must be treated like strings and have
+# the appropriate number of preceeding zeros
+#
+# stagedir The directory location of the reference case files.
+# ==============================================================================
+
+setenv refcase cesm_hybrid
+setenv refyear 2004
+setenv refmon 01
+setenv refday 10
+setenv reftod 00000
+
+# useful combinations of time that we use below
+setenv refdate $refyear-$refmon-$refday
+setenv reftimestamp $refyear-$refmon-$refday-$reftod
+
+setenv stagedir /glade/p/image/CESM_initial_ensemble/rest/${reftimestamp}
+# alternative reference case for different times may be available here:
+# setenv stagedir /glade/scratch/${USER}/${refcase}/archive/rest/${reftimestamp}
+# or on the HPSS:
+# /CCSM/dart/FV0.9x1.25x30_cesm1_1_1/{Mon}1 for 1-degree FV ensembles
+
+# ==============================================================================
+# runtime settings: This script will find usable files for years 19mumble-2010.
+# Years after that (or before) may require searching $cesmdata for more
+# up-to-date files and adding them to the user_nl_cam_#### in the code below.
+#
+# start_year generally this is the same as the reference case date, but it can
+# start_month be different if you want to start this run as if it was a different time.
+# start_day
+# start_tod
+#
+# sst_use_defaults Controls what data ocean files are used.
+# 'true' makes CESM use default files,
+# 'false' requires you to supply a set of files
+# sst_dataset Data ocean file
+# sst_grid Supporting (consistent) grid file
+# sst_year_start Years included in the sst files.
+# sst_year_end
+# The default SST (as of 2015-3) goes through 2012.
+# Don't use the last few months, since they are incomplete.
+#
+# short_term_archiver Copies the files from each job step to a 'rest' directory.
+# long_term_archiver Puts the files from all completed steps on tape storage.
+#
+# resubmit How many job steps to run on continue runs (should be 0 initially)
+# stop_option Units for determining the forecast length between assimilations
+# stop_n Number of time units in each forecast
+#
+# If the long-term archiver is off, you get a chance to examine the files before
+# they get moved to long-term storage. You can always submit $CASE.l_archive
+# whenever you want to free up space in the short-term archive directory.
+# ==============================================================================
+
+setenv start_year 2004
+setenv start_month 01
+setenv start_day 10
+setenv start_tod 00000
+
+setenv sst_use_defaults 'true'
+
+if ($sst_use_defaults == 'false') then
+ setenv sst_dataset ${cesmdata}/atm/cam/sst/sst_HadOIBl_bc_0.9x1.25_1850_2013_c140701.nc
+ setenv sst_grid ${cesmdata}/share/domains/domain.ocn.fv0.9x1.25_gx1v6.130409.nc
+ setenv sst_year_start 1850
+ setenv sst_year_end 2011
+endif
+
+# Cycles; all archiving is handled outside of CESM.
+# Both variables should be 'off'.
+setenv short_term_archiver off
+setenv long_term_archiver off
+
+# Cycles; reset RESUBMIT in ${case}.submit_cycles, not here, resubmit = 0.
+setenv resubmit 0
+setenv stop_option nhours
+setenv stop_n 6
+
+# ==============================================================================
+# job settings:
+#
+# queue can be changed during a series by changing the ${case}.run
+# timewall can be changed during a series by changing the ${case}.run
+#
+# TJH: Advancing 30 instances for 6 hours and assimilating took
+# less than 10 minutes on yellowstone using 1800 pes (120 nodes)
+# ==============================================================================
+
+setenv ACCOUNT NIMGxxxx
+setenv queue economy
+setenv timewall 0:20
+
+# ==============================================================================
+# standard commands:
+#
+# If you are running on a machine where the standard commands are not in the
+# expected location, add a case for them below.
+# ==============================================================================
+
+set nonomatch # suppress "rm" warnings if wildcard does not match anything
+
+# The FORCE options are not optional.
+# The VERBOSE options are useful for debugging though
+# some systems don't like the -v option to any of the following
+switch ("`hostname`")
+ case be*:
+ # NCAR "bluefire"
+ set MOVE = '/usr/local/bin/mv -fv'
+ set COPY = '/usr/local/bin/cp -fv --preserve=timestamps'
+ set LINK = '/usr/local/bin/ln -fvs'
+ set REMOVE = '/usr/local/bin/rm -fr'
+
+ breaksw
+ default:
+ # NERSC "hopper", NWSC "yellowstone"
+ set MOVE = '/bin/mv -fv'
+ set COPY = '/bin/cp -fv --preserve=timestamps'
+ set LINK = '/bin/ln -fvs'
+ set REMOVE = '/bin/rm -fr'
+
+ breaksw
+endsw
+
+# ==============================================================================
+# ==============================================================================
+# by setting the values above you should be able to execute this script and
+# have it run. however, for running a real experiment there are still many
+# settings below this point - e.g. component namelists, history file options,
+# the processor layout, xml file options, etc - that you will almost certainly
+# want to change before doing a real science run.
+# ==============================================================================
+# ==============================================================================
+
+
+# ==============================================================================
+# Make sure the CESM directories exist.
+# VAR is the shell variable name, DIR is the value
+# ==============================================================================
+
+foreach VAR ( cesmroot dartroot stagedir )
+ set DIR = `eval echo \${$VAR}`
+ if ( ! -d $DIR ) then
+ echo "ERROR: directory '$DIR' not found"
+ echo " In the setup script check the setting of: $VAR "
+ exit -1
+ endif
+end
+
+# ==============================================================================
+# Create the case - this creates the CASEROOT directory.
+#
+# For list of the pre-defined component sets: ./create_newcase -list
+# To create a variant compset, see the CESM documentation and carefully
+# incorporate any needed changes into this script.
+# ==============================================================================
+
+# fatal idea to make caseroot the same dir as where this setup script is
+# since the build process removes all files in the caseroot dir before
+# populating it. try to prevent shooting yourself in the foot.
+
+if ( $caseroot == `dirname $0` ) then
+ echo "ERROR: the setup script should not be located in the caseroot"
+ echo "directory, because all files in the caseroot dir will be removed"
+ echo "before creating the new case. move the script to a safer place."
+ exit -1
+endif
+
+echo "removing old files from ${caseroot}"
+echo "removing old files from ${exeroot}"
+echo "removing old files from ${rundir}"
+${REMOVE} ${caseroot}
+${REMOVE} ${exeroot}
+${REMOVE} ${rundir}
+
+${cesmroot}/scripts/create_newcase -case ${caseroot} -mach ${mach} \
+ -res ${resolution} -compset ${compset}
+
+if ( $status != 0 ) then
+ echo "ERROR: Case could not be created."
+ exit -1
+endif
+
+# Preserve a copy of this script as it was run.
+set ThisFileName = $0:t
+${COPY} $ThisFileName ${caseroot}/${ThisFileName}.original
+
+# ==============================================================================
+# Record the DARTROOT directory and copy the DART setup script to CASEROOT.
+# CESM_DART_config can be run at some later date if desired, but it presumes
+# to be run from a CASEROOT directory. If CESM_DART_config does not exist locally,
+# then it better exist in the expected part of the DARTROOT tree.
+# ==============================================================================
+
+if ( ! -e CESM_DART_config ) then
+ ${COPY} ${dartroot}/models/cam/shell_scripts/CESM_DART_config .
+endif
+
+if ( -e CESM_DART_config ) then
+ sed -e "s#BOGUS_DART_ROOT_STRING#${dartroot}#" < CESM_DART_config >! temp.$$
+ ${MOVE} temp.$$ ${caseroot}/CESM_DART_config
+ chmod 755 ${caseroot}/CESM_DART_config
+else
+ echo "ERROR: the script to configure for data assimilation is not available."
+ echo " CESM_DART_config MUST be present locally or in"
+ echo " ${dartroot}/models/cam/shell_scripts/"
+ exit -2
+endif
+
+# Cycles; set up scripts for cycles archiving.
+if (-e archive_cycles.csh) then
+ set ACCNT = `grep BSUB archive_cycles.csh | grep -e '-P' `
+ sed -e "s#BOGUS_CASE_DIRECTORY#$caseroot#" \
+ -e "/-P/s#$ACCNT[3]#$ACCOUNT#" \
+ < archive_cycles.csh >! $caseroot/archive_cycles.csh
+else
+ echo "WARNING: No archive_cycles.csh available. "
+ echo "WARNING: This means YOU are responsible for archiving."
+ echo " Your run directory will fill up fast. Pay Attention!"
+endif
+
+# ==============================================================================
+# Configure the case.
+# ==============================================================================
+
+cd ${caseroot}
+
+source ./Tools/ccsm_getenv || exit -2
+
+# Make sure the case is configured with a data ocean.
+
+if ( ${COMP_OCN} != docn ) then
+ echo " "
+ echo "ERROR: This setup script is not appropriate for active ocean compsets."
+ echo "ERROR: Please use the models/CESM/shell_scripts examples for that case."
+ echo " "
+ exit -3
+endif
+
+# MAX_TASKS_PER_NODE comes from $case/Tools/mkbatch.$machine
+@ ptile = $MAX_TASKS_PER_NODE / 2
+@ nthreads = 1
+
+# Save a copy for debug purposes
+foreach FILE ( *xml )
+ if ( ! -e ${FILE}.original ) then
+ ${COPY} $FILE ${FILE}.original
+ endif
+end
+
+# NOTE: If you require bit-for-bit agreement between different runs,
+# in particular, between pmo (single instance) and assimilations (NINST > 1),
+# or if you need to change the number of nodes/member due to changing memory needs,
+# then env_run.xml:BFBFLAG must be set to TRUE, so that the coupler will
+# generate bit-for-bit identical results, regardless of the number of tasks
+# given to it. The time penalty appears to be ~ 0.5% in the forecast.
+# Alternatively, you can set cpl_tasks = same_number in both experiments
+
+# Task layout:
+# Set the nodes_per_instance below to match your case. If you get 'out of memory'
+# errors OR failures without any messages, try increasing the nodes_per_instance.
+# CAM-FV 1 degree can run on 2 nodes/instance on yellowstone.
+# CAM-SE ne30 (~ 1 degree) needed 5 nodes/instance.
+# By computing task counts like we do below, we guarantee each instance uses
+# a whole number of nodes which is the recommended configuration.
+
+# Yellowstone: no large memory nodes, and 15 tasks/node is recommended.
+# Edwards says there's no speed up by running non-active components concurrently,
+# after ATM has run, so just run all components sequentially.
+# BUT, do arrange it so that each member(instance) spans complete nodes:
+# modulo(total pe count / number of instances, 15) == 0.
+
+@ nodes_per_instance = 2
+
+@ atm_tasks = $ptile * $num_instances * $nodes_per_instance
+@ lnd_tasks = $ptile * $num_instances * $nodes_per_instance
+@ ice_tasks = $ptile * $num_instances * $nodes_per_instance
+@ ocn_tasks = $ptile * $num_instances
+@ cpl_tasks = $ptile * $num_instances
+@ glc_tasks = $ptile * $num_instances
+@ rof_tasks = $ptile * $num_instances
+@ wav_tasks = $ptile * $num_instances
+
+
+switch ("`hostname`")
+ case ys*:
+ if ($atm_tasks > 4100) then
+ echo 'WARNING! Running DART on yellowstone with > 4100 tasks can cause filter'
+ echo ' to hang in MPI_finalize. Either reduce your task request,'
+ echo ' or contact DART for a temporary work-around.'
+ exit -9
+ endif
+ breaksw
+endsw
+
+echo ""
+echo "ATM gets $atm_tasks"
+echo "LND gets $lnd_tasks"
+echo "ICE gets $ice_tasks"
+echo "OCN gets $ocn_tasks"
+echo "CPL gets $cpl_tasks"
+echo "GLC gets $glc_tasks"
+echo "ROF gets $rof_tasks"
+echo "WAV gets $wav_tasks"
+echo ""
+
+./xmlchange NTHRDS_ATM=$nthreads,NTASKS_ATM=$atm_tasks,NINST_ATM=$num_instances
+./xmlchange NTHRDS_LND=$nthreads,NTASKS_LND=$lnd_tasks,NINST_LND=$num_instances
+./xmlchange NTHRDS_ICE=$nthreads,NTASKS_ICE=$ice_tasks,NINST_ICE=$num_instances
+./xmlchange NTHRDS_OCN=$nthreads,NTASKS_OCN=$ocn_tasks,NINST_OCN=1
+./xmlchange NTHRDS_CPL=$nthreads,NTASKS_CPL=$cpl_tasks
+./xmlchange NTHRDS_GLC=$nthreads,NTASKS_GLC=$glc_tasks,NINST_GLC=1
+./xmlchange NTHRDS_ROF=$nthreads,NTASKS_ROF=$rof_tasks,NINST_ROF=1
+./xmlchange NTHRDS_WAV=$nthreads,NTASKS_WAV=$wav_tasks,NINST_WAV=1
+./xmlchange ROOTPE_ATM=0
+./xmlchange ROOTPE_LND=0
+./xmlchange ROOTPE_ICE=0
+./xmlchange ROOTPE_OCN=0
+./xmlchange ROOTPE_CPL=0
+./xmlchange ROOTPE_GLC=0
+./xmlchange ROOTPE_ROF=0
+./xmlchange ROOTPE_WAV=0
+
+# http://www.cesm.ucar.edu/models/cesm1.2/cesm/doc/usersguide1_2/c1096.html#run_start_stop
+# "A hybrid run indicates that CESM is initialized more like a startup, but uses
+# initialization datasets from a previous case. This is somewhat analogous to a
+# branch run with relaxed restart constraints. A hybrid run allows users to bring
+# together combinations of initial/restart files from a previous case (specified
+# by $RUN_REFCASE) at a given model output date (specified by $RUN_REFDATE).
+# Unlike a branch run, the starting date of a hybrid run (specified by $RUN_STARTDATE)
+# can be modified relative to the reference case. In a hybrid run, the model does not
+# continue in a bit-for-bit fashion with respect to the reference case. The resulting
+# climate, however, should be continuous provided that no model source code or
+# namelists are changed in the hybrid run. In a hybrid initialization, the ocean
+# model does not start until the second ocean coupling (normally the second day),
+# and the coupler does a "cold start" without a restart file."
+#
+# TJH:
+# DART's CAM implementation causes a bit more complexity. DART only uses CAM _initial_
+# files, not RESTART files, so there are sourcemods to force a hybrid start for CAM to
+# read initial files - even when CONTINUE_RUN = TRUE.
+
+./xmlchange RUN_TYPE=hybrid
+./xmlchange RUN_STARTDATE=${start_year}-${start_month}-${start_day}
+./xmlchange START_TOD=$start_tod
+./xmlchange RUN_REFCASE=$refcase
+./xmlchange RUN_REFDATE=$refdate
+./xmlchange RUN_REFTOD=$reftod
+./xmlchange BRNCH_RETAIN_CASENAME=FALSE
+./xmlchange GET_REFCASE=FALSE
+./xmlchange EXEROOT=${exeroot}
+./xmlchange RUNDIR=${rundir}
+
+if ($sst_use_defaults == 'false') then
+ ./xmlchange SSTICE_DATA_FILENAME=$sst_dataset
+ ./xmlchange SSTICE_GRID_FILENAME=$sst_grid
+ ./xmlchange SSTICE_YEAR_ALIGN=$sst_year_start
+ ./xmlchange SSTICE_YEAR_START=$sst_year_start
+ ./xmlchange SSTICE_YEAR_END=$sst_year_end
+endif
+
+# Do not change the CALENDAR or the value of CONTINUE_RUN in this script.
+
+./xmlchange CALENDAR=GREGORIAN
+./xmlchange CONTINUE_RUN=FALSE
+
+./xmlchange STOP_OPTION=$stop_option
+./xmlchange STOP_N=$stop_n
+./xmlchange RESUBMIT=$resubmit
+
+./xmlchange PIO_TYPENAME=pnetcdf
+
+set TEST_MPI = `./xmlquery -valonly MPI_RUN_COMMAND | sed -e 's/MPI_RUN_COMMAND = //'`
+if (${TEST_MPI} == 'UNSET') then
+ ./xmlchange MPI_RUN_COMMAND=mpirun.lsf
+endif
+
+# The river transport model ON is useful only when using an active ocean or
+# land surface diagnostics. Setting ROF_GRID, RTM_MODE to 'null' turns off the RTM.
+# If you turn it ON, you will have to stage initial files etc.
+
+./xmlchange ROF_GRID='null'
+./xmlchange RTM_MODE='NULL'
+
+# COUPLING discussion. F compsets are 'tight' coupling.
+# Only change the ATM_NCPL ... everything is based on this one value,
+# including CAM physics and dynamics timesteps.
+# Default values for coupling are preserved in env_run.xml.original
+
+./xmlchange NCPL_BASE_PERIOD=day
+./xmlchange ATM_NCPL=48
+
+# These are archiving options that may be used.
+# You can turn the short/long term archivers on or off,
+# but these settings should be made in either event.
+
+./xmlchange DOUT_S_ROOT=${archdir}
+./xmlchange DOUT_S_SAVE_INT_REST_FILES=FALSE
+./xmlchange DOUT_L_MSROOT="csm/${case}"
+./xmlchange DOUT_L_HTAR=FALSE
+
+if ($short_term_archiver == 'off') then
+ ./xmlchange DOUT_S=FALSE
+else
+ ./xmlchange DOUT_S=TRUE
+endif
+if ($long_term_archiver == 'off') then
+ ./xmlchange DOUT_L_MS=FALSE
+else
+ ./xmlchange DOUT_L_MS=TRUE
+endif
+
+# DEBUG = TRUE implies turning on run and compile time debugging.
+# INFO_DBUG level of debug output, 0=minimum, 1=normal, 2=more, 3=too much.
+# WARNING: CAM-SE fails if DEBUG=TRUE
+./xmlchange DEBUG=FALSE
+./xmlchange INFO_DBUG=0
+
+# ==============================================================================
+# Update source files.
+# Ideally, using DART would not require any modifications to the model source.
+# Until then, this script accesses sourcemods from a hardwired location.
+# If you have additional sourcemods, they will need to be merged into any DART
+# mods and put in the SourceMods subdirectory found in the 'caseroot' directory.
+# ==============================================================================
+
+if ( ! -d ~/${cesmtag}/SourceMods ) then
+ echo "ERROR - No SourceMods for this case."
+ echo "ERROR - No SourceMods for this case."
+ echo "DART requires modifications to several src files."
+ echo "Download the appropriate files for "$cesmtag" from:"
+ echo "http://www.image.ucar.edu/pub/DART/CESM"
+ echo "untar these into your HOME directory - they will create a"
+ echo "~/cesm1_2_1 directory with the appropriate SourceMods structure."
+ exit -4
+endif
+
+# Copy all of the 'generic' SourceMods
+${COPY} -r ~/${cesmtag}/SourceMods/* ${caseroot}/SourceMods/ || exit 2
+
+# Each CLM version has some SourceMods. Link to the right version.
+# must parse from a variable of the form:
+# CLM_CONFIG_OPTS: -phys clm4_0 -bgc cn
+
+set clm_opts = `echo $CLM_CONFIG_OPTS | sed -e "s/-//"`
+
+@ iarg = 1
+while ($iarg <= $#clm_opts)
+
+ @ iargp1 = $iarg + 1
+ set option = $clm_opts[$iarg]
+ set value = $clm_opts[$iargp1]
+
+ switch ( ${option} )
+ case "phys":
+ if ( -e SourceMods/src.clm/src/${value} ) then
+ cd SourceMods/src.clm
+ ${LINK} src/${value}/*/*F90 .
+ cd ../..
+ else
+ echo "No SourceMods for CLM <${value}>."
+ echo "Got the version from CLM_CONFIG_OPTS ... <${CLM_CONFIG_OPTS}>"
+ endif
+ breaksw
+ # case "bgc": no special action needed here at this time
+ # breaksw
+
+ default:
+ breaksw
+ endsw
+
+ @ iarg = $iarg + 2
+end
+
+# Need to know if we are using WACCM (aka WCCM) for several reasons.
+# Mostly file managment issues.
+set waccm = `echo $CCSM_COMPSET | grep CCM`
+if ($status == 0) then
+ set waccm = true
+else
+ set waccm = false
+ # WACCM benefits from a modified cd_core.F90, but none of the
+ # other configurations do.
+ echo "Using the default cd_core.F90"
+ ${REMOVE} -v SourceMods/src.cam/src/dynamics/${CAM_DYCORE}/cd_core.F90
+endif
+
+# Each CAM dynamical core has its own SourceMods
+if ( -e SourceMods/src.cam/src/dynamics/${CAM_DYCORE}/*F90 ) then
+ cd SourceMods/src.cam
+ ${LINK} src/dynamics/${CAM_DYCORE}/*F90 .
+ cd ../..
+else
+ echo "No SourceMods for CAM dycore <${CAM_DYCORE}>."
+endif
+
+# The CESM multi-instance capability is relatively new and still has a few
+# implementation bugs. These are known problems and will be fixed soon.
+# this should be removed when the files are fixed:
+
+echo "REPLACING BROKEN CESM FILES HERE - SHOULD BE REMOVED WHEN FIXED"
+echo caseroot is ${caseroot}
+if ( -d ~/${cesmtag} ) then
+
+ # preserve the original version of the files
+ if ( ! -e ${caseroot}/Buildconf/rtm.buildnml.csh.original ) then
+ ${COPY} ${caseroot}/Buildconf/rtm.buildnml.csh \
+ ${caseroot}/Buildconf/rtm.buildnml.csh.original
+ endif
+
+ # patch/replace the broken files (first used by cesm_setup)
+ ${COPY} ~/${cesmtag}/rtm.buildnml.csh ${caseroot}/Buildconf/.
+
+endif
+
+# ==============================================================================
+# Set up the case.
+# This creates the EXEROOT and RUNDIR directories.
+# ==============================================================================
+
+echo 'Setting up the case ...'
+
+./cesm_setup
+
+if ( $status != 0 ) then
+ echo "ERROR: Case could not be set up."
+ exit -2
+endif
+
+# ==============================================================================
+# Edit the run script to reflect queue and wallclock
+# ==============================================================================
+
+echo ''
+echo 'Updating the run script to set wallclock and queue.'
+echo ''
+
+if ( ! -e ${case}.run.original ) then
+ ${COPY} ${case}.run ${case}.run.original
+endif
+
+# Is this redundant with the one above? Or are there new values?
+source Tools/ccsm_getenv
+set BATCH = `echo $BATCHSUBMIT | sed 's/ .*$//'`
+switch ( $BATCH )
+ case bsub*:
+ # NCAR "bluefire", "yellowstone"
+ set TIMEWALL=`grep BSUB ${case}.run | grep -e '-W' `
+ set QUEUE=`grep BSUB ${case}.run | grep -e '-q' `
+ sed -e "/BSUB/s#$TIMEWALL[3]#$timewall#" \
+ -e "/BSUB/s#ptile=[0-9][0-9]*#ptile=$ptile#" \
+ -e "/BSUB/s#$QUEUE[3]#$queue#" < ${case}.run >! temp.$$
+ ${MOVE} temp.$$ ${case}.run
+ chmod 755 ${case}.run
+ breaksw
+
+ default:
+ breaksw
+endsw
+
+# This is the part that modifies the run script to allow CESM to advance
+# correctly given the modifications of CAM for DART. It also copies several
+# required DART files to the caseroot directory.
+
+./CESM_DART_config || exit -3
+
+
+# ==============================================================================
+# Cycles;
+# Make a cycles version of the .run script, AFTER all changes to $case.run have been made.
+# ==============================================================================
+cat >! temp.run_cycles << EndWarn
+ HEY! This is a script intended only for advanced users
+ on NCAR\'s yellowstone computer.
+ Please consult us before using it, since it does not have the level of
+ error checking and commenting which other DART scripts have.
+EndWarn
+
+# We're in $caseroot
+# Replace the command to submit the next job to the queue
+# with a command to run the next forecast+assimilation in the current job.
+# The '|| exit 33' makes the whole recursive cascade return an error flag to the LSF job.
+# Some other mechanism might work.
+# Single quotes prevent expansion of BATCHSUBMIT and CASE.
+sed -e 's#$BATCHSUBMIT ./$CASE.run#./$CASE.run_cycles || exit 33#' \
+ < $case.run >> temp.run_cycles
+
+# Insert section of script to query RESUBMIT from env_run.xml and,
+# if it = -1, set it to the value from ${case}.submit_cycles
+set file_len = `wc -l temp.run_cycles`
+set insert_line = `grep -n 'cd $CASE' temp.run_cycles | head -n 1 | sed -e "s#:# #" `
+@ insert1 = $insert_line[1]
+head -n $insert1 temp.run_cycles >! ${case}.run_cycles
+
+cat >> ${case}.run_cycles <<EndSect1
+
+ # Conditionally change RESUBMIT back to the starting value
+ # when a new cycles job is starting.
+ # This is done for the first cycles job by $case.submit_cycles.
+ echo "RESUBMIT = \$RESUBMIT"
+ if (\$RESUBMIT == -1) then
+ set line = \`grep 'set RESUBMIT' ${caseroot}/${case}.submit_cycles\`
+ echo "Resetting RESUBMIT to \$line[4]"
+ ./xmlchange RESUBMIT=\$line[4]
+ endif
+
+ # Change CONTINUE_RUN from its startup value.
+ ./xmlchange CONTINUE_RUN=TRUE
+
+EndSect1
+
+
+# Insert code which handles the RESUBMIT = 0 case,
+# when we want to stop this job and signal the next that it needs to reset RESUBMIT.
+
+set insert_line = `grep -n 'problem sourcing tempres' temp.run_cycles | sed -e "s#:# #" `
+@ insert2 = $insert_line[1]
+@ next_exist_section = $insert2 - $insert1
+head -n $insert2 temp.run_cycles | tail -n $next_exist_section >> ${case}.run_cycles
+
+cat >> ${case}.run_cycles <<EndSect2
+ exit 111
+ else
+ rm tempres
+ endif
+ endif
+else
+ # Don't run another $case.run_cycles, and signal next job that RESUBMIT needs to be reset.
+ @ RESUBMIT = \$RESUBMIT - 1
+ echo "Signal next job to start with new RESUBMIT. RESUBMIT is now \$RESUBMIT".
+ ./xmlchange RESUBMIT=\$RESUBMIT
+EndSect2
+
+# The rest of the run_cycles file
+# 3 is for the 'rm temp_res' and 2 'endif' lines which were there, and are being replaced.
+@ rest_of_file = $file_len[1] - ($insert2 + 3)
+tail -n $rest_of_file temp.run_cycles >> ${case}.run_cycles
+chmod 755 ${case}.run_cycles
+$REMOVE temp.run_cycles
+
+# ==============================================================================
+# Modify namelist templates for each instance.
+#
+# In a hybrid run with CONTINUE_RUN = FALSE (i.e. just starting up):
+#
+# CAM has been forced to read initial files - specified by namelist var:ncdata
+# CICE reads from namelist variable 'ice_ic'
+# CLM builds its own 'finidat' value from the REFCASE variables but in CESM1_2_1
+# it does not use the instance string. There is a patch for clm.buildnml.csh
+#
+# When CONTINUE_RUN = TRUE, CICE and CLM read from pointer files.
+#
+# All of these must later on be staged with these same filenames.
+# OR - all these namelists can be changed to match whatever has been staged.
+# MAKE SURE THE STAGING SECTION OF THIS SCRIPT MATCHES THESE VALUES.
+# ==============================================================================
+
+@ inst = 1
+while ($inst <= $num_instances)
+
+ # following the CESM strategy for 'inst_string'
+ set inst_string = `printf _%04d $inst`
+
+ # ===========================================================================
+ set fname = "user_nl_cam${inst_string}"
+ # ===========================================================================
+ # ATM Namelist
+ # DART/CAM requires a PHIS field in a history file at the same frequency as
+ # the CAM assimilation. This script sets it to be written to the .h0. every
+ # assimilation time. If you want to write it to a different .h?. file, you MUST
+ # modify the assimilate.csh script in TWO places. You will need to set
+ # 'empty_htapes = .false.' and change 'nhtfrq' and 'mfilt' to get a CAM
+ # default-looking .h0. file.
+ #
+ # inithist == 'ENDOFRUN' ensures that CAM writes the required initial file
+ # every time it stops.
+
+ echo " inithist = 'ENDOFRUN'" >> ${fname}
+ echo " ncdata = 'cam_initial${inst_string}.nc'" >> ${fname}
+ echo " empty_htapes = .true. " >> ${fname}
+ echo " fincl1 = 'PHIS:I' " >> ${fname}
+ echo " nhtfrq = -$stop_n " >> ${fname}
+ echo " mfilt = 1 " >> ${fname}
+
+ # CAM forcing files.
+ # Some of the files specified here are because the default files only
+ # contain data through 2005 and we are interested in timeframes after that.
+
+ # CAM5 does prognostic aerosols by default (which is expensive).
+ # Specifying the following causes CAM to use prescribed aerosols, which is cheaper.
+ # Prescribed aerosols on the timescales of interest to assimilation are acceptable.
+ # WACCM5 should not have these set.
+
+ if ($waccm == false ) then
+
+ set chem_datapath = "${cesmdata}/atm/cam/chem/trop_mozart_aero"
+
+ echo "prescribed_ozone_file = 'ozone_1.9x2.5_L26_1850-2015_rcp45_c101108.nc'" >> ${fname}
+ echo "tracer_cnst_file = 'oxid_1.9x2.5_L26_1850-2015_rcp45_c101108.nc'" >> ${fname}
+
+ echo "aerodep_flx_datapath = '${chem_datapath}/aero'" >> ${fname}
+ echo "aerodep_flx_file = 'aerosoldep_rcp4.5_monthly_1849-2104_0.9x1.25_c100407.nc'" >> ${fname}
+ echo "aerodep_flx_type = 'CYCLICAL'" >> ${fname}
+ echo "aerodep_flx_cycle_yr = 2000" >> ${fname}
+
+ if ($start_year > 2005 && $start_year <= 2010) then
+ # As of April 2015, the default files only run through 2005.
+ # Specifying non-default files for assimilation for years 2006-2010.
+ # Update as needed and newer files become available.
+
+ echo " ext_frc_specifier = " >> ${fname}
+ echo " 'SO2 -> ${chem_datapath}/emis/ar5_mam3_so2_elev_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'bc_a1 -> ${chem_datapath}/emis/ar5_mam3_bc_elev_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'num_a1 -> ${chem_datapath}/emis/ar5_mam3_num_a1_elev_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'num_a2 -> ${chem_datapath}/emis/ar5_mam3_num_a2_elev_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'pom_a1 -> ${chem_datapath}/emis/ar5_mam3_oc_elev_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'so4_a1 -> ${chem_datapath}/emis/ar5_mam3_so4_a1_elev_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'so4_a2 -> ${chem_datapath}/emis/ar5_mam3_so4_a2_elev_1850-2010_c20100902_v12.nc'" >> ${fname}
+
+ echo " srf_emis_specifier = " >> ${fname}
+ echo " 'DMS -> ${chem_datapath}/emis/aerocom_mam3_dms_surf_1849-2010_c20100902.nc'," >> ${fname}
+ echo " 'SO2 -> ${chem_datapath}/emis/ar5_mam3_so2_surf_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'SOAG -> ${chem_datapath}/emis/ar5_mam3_soag_1.5_surf_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'bc_a1 -> ${chem_datapath}/emis/ar5_mam3_bc_surf_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'num_a1 -> ${chem_datapath}/emis/ar5_mam3_num_a1_surf_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'num_a2 -> ${chem_datapath}/emis/ar5_mam3_num_a2_surf_1850-2010_c20100902_v12.nc'," >> ${fname}
+ echo " 'pom_a1 -> ${chem_datapath}/emis/ar5_mam3_oc_surf_1850-2010_c20100902_v12.nc'," >> ${fname}
@@ Diff output truncated at 40000 characters. @@
More information about the Dart-dev
mailing list