[Dart-dev] DART/branches Revision: 12603
dart at ucar.edu
dart at ucar.edu
Fri May 25 16:24:43 MDT 2018
raeder at ucar.edu
2018-05-25 16:24:43 -0600 (Fri, 25 May 2018)
2531
The context for this commit is that extensive work has been completed
in CESM, CIME, and DART, to enable CAM assimilations using every
CESM2_0 and later release (and even beta-tag).
In addition, the cam-fv model_mod has been completely rewritten
under the RMA architecture. The major changes are listed here.
DART's CESM SourceMods have been integrated into CESM.
All known multi-instance, multi-driver, large ensemble bugs in CIME
have been fixed. Using the multi-driver capability is now the default,
because it saves >60% of forecast core hours and wall clock for an
80 member ensemble.
The CESM file naming convention has been completely implemented in
assimilate.csh for all CAM+DART files which need to be archived.
This required the creation of a $case.dart.r file, which is needed
to hold the names of the inflation "restart history" files
($case.dart.rh.*inf*).
CIME's archive mechanism (st_archive) has been updated to handle
all CAM+DART files. There is no longer a CESM long term archiver
(lt_archive).
The initial inflation restart file mechanism now uses fill_inflation_restart
in assimilate.csh, instead of the inflation cookie in stage_cesm_files.
Inflation restart files will be created if needed for the first cycle.
Otherwise, files provided by the user will be used.
The setup scripts use a new CAM compset (FHIST_DARTC6) by default.
It was designed to include minimal "physics" that has little influence
on an atmospheric assimilation. It is used in a new CESM pre-beta test
of capabilities needed by DART. Assimilations for scientific purposes
may need a more standard compset, which will be identified and
implemented later.
The setup scripts use the default solar, greenhouse gas, aerosol, ...
forcing files. This will probably need to be changed to assimilate in
recent years.
The setup scripts use the functioning, CLM, initial restart file,
interpolation mechanism, so that an assimilation can use spun up model
states from a different CLM tag/release than used for the assimilation.
They have been updated to run on cheyenne (PBS) as well as LSF.
Case_st_archive.py no longer needs modification by DART
and has been removed.
The final form of these scripts have been tested using a 2 degree,
3 member ensemble perturbed from a single member and assimilated NCEP+ACARS.
The tests are named CESM2_0_test1 and 2. Earlier tests used a larger
ensemble and higher resolution (larger state vector).
This code should be reviewed soon and before recam is merged onto the
rma_trunk.
Modified: DART/branches/recam/models/cam-fv/shell_scripts/cesm2_0/assimilate.csh
===================================================================
--- DART/branches/recam/models/cam-fv/shell_scripts/cesm2_0/assimilate.csh 2018-05-25 17:23:52 UTC (rev 12602)
+++ DART/branches/recam/models/cam-fv/shell_scripts/cesm2_0/assimilate.csh 2018-05-25 22:24:43 UTC (rev 12603)
@@ -12,8 +12,6 @@
# See 'RMA' for places where there are questions about RMA,
# especially file naming.
-
-
#=========================================================================
# Block 0: Set command environment
#=========================================================================
@@ -31,10 +29,38 @@
# "Fix" that here, so the rest of the script isn't confusing.
@ cycle = $2 + 1
+# Tag the state output with CESM format names:
+# ${case}.${scomp}[_$inst}.${filetype}[.dart_file].${date}.nc
+# These should all be named with $scomp = "cam" to distinguish
+# them from the same output from other components in multi-component assims.
+# The file type "i" or "e.$dart_file" will distinguish CAM files from DART.
+set scomp = "cam"
+
+# In CESM1_4 xmlquery must be executed in $CASEROOT.
+cd ${CASEROOT}
+setenv CASE $CASEROOT:t
+setenv ensemble_size `./xmlquery NINST_ATM --value`
+setenv CAM_DYCORE `./xmlquery CAM_DYCORE --value`
+setenv EXEROOT `./xmlquery EXEROOT --value`
+setenv RUNDIR `./xmlquery RUNDIR --value`
+setenv archive `./xmlquery DOUT_S_ROOT --value`
+setenv TOTALPES `./xmlquery TOTALPES --value`
+setenv CONT_RUN `./xmlquery CONTINUE_RUN --value`
+setenv DATA_ASSIMILATION_CYCLES `./xmlquery DATA_ASSIMILATION_CYCLES --value`
+cd $RUNDIR
+
+# A switch to save all the inflation files
+setenv save_all_inf TRUE
+# A switch to signal how often to save the stages' ensemble members: NONE, RESTART_TIMES, ALL
+# Mean and sd will always be saved.
+setenv save_stages_freq RESTART_TIMES
+
+set BASEOBSDIR = BOGUSBASEOBSDIR
+
# The FORCE options are not optional.
# The VERBOSE options are useful for debugging though
# some systems don't like the -v option to any of the following
-switch ("`hostname`")
+switch ($HOSTNAME)
case ys*:
# NCAR "yellowstone"
set MOVE = 'mv -fv'
@@ -43,21 +69,36 @@
set REMOVE = 'rm -fr'
set TASKS_PER_NODE = `echo $LSB_SUB_RES_REQ | sed -ne '/ptile/s#.*\[ptile=\([0-9][0-9]*\)]#\1#p'`
setenv MP_DEBUG_NOTIMEOUT yes
-
- set BASEOBSDIR = BOGUSBASEOBSDIR
set LAUNCHCMD = mpirun.lsf
- breaksw
+ breaksw
+ case ch*:
+ # NCAR "cheyenne" -v removed because it doesn't work on (my) cheyenne.
+ # Kluge to make environment match what cesm2_0_beta07+cime5.3.0-alpha.01 (2017-11-21)
+# Turn off for first test of CIME_DA_vars_4 to see if this is resolved by default.
+# It's not as of 2018-1-25
+ module switch mpt mpt/2.15
+ set MOVE = 'mv -f'
+ set COPY = 'cp -f --preserve=timestamps'
+ set LINK = 'ln -fs'
+ set LIST = '/usr/bin/ls '
+ set REMOVE = 'rm -fr'
+ # echo "Trying to set TASKS_PER_NODE using PBS_NUM_PPN $PBS_NUM_PPN"
+ # Unavailable for some reason: set TASKS_PER_NODE = $PBS_NUM_PPN
+ set TASKS_PER_NODE = 36
+ setenv MP_DEBUG_NOTIMEOUT yes
+ set LAUNCHCMD = mpiexec_mpt
+ breaksw
+
case linux_system_with_utils_in_other_dirs*:
# example of pointing this script at a different set of basic commands
set MOVE = '/usr/local/bin/mv -fv'
set COPY = '/usr/local/bin/cp -fv --preserve=timestamps'
set LINK = '/usr/local/bin/ln -fvs'
+ set LIST = '/usr/local/bin/ls '
set REMOVE = '/usr/local/bin/rm -fr'
-
- set BASEOBSDIR = BOGUSBASEOBSDIR
set LAUNCHCMD = mpirun.lsf
- breaksw
+ breaksw
default:
# NERSC "hopper"
@@ -65,32 +106,105 @@
set COPY = 'cp -fv --preserve=timestamps'
set LINK = 'ln -fvs'
set REMOVE = 'rm -fr'
+ set LAUNCHCMD = "aprun -n $TOTALPES"
More information about the Dart-dev
mailing list