[Dart-dev] DART/branches Revision: 13191
dart at ucar.edu
dart at ucar.edu
Tue Jul 9 16:15:00 MDT 2019
raeder at ucar.edu
2019-07-09 16:15:00 -0600 (Tue, 09 Jul 2019)
3317
Lots of fixes resulting from testing repack_st_archive.csh and purge.csh.
This has been tested from setup through purge in a 3 member, 2 day test.
These changes have not been reviewed by anyone.
They are not entirely compliant with DART code conventions.
launch_cf.sh
> Running under slurm requires the use of a different task counter environment variable.
This new local version handles both PBS and slurm.
cam-fv scripts will use it instead of any system default version.
mv_to_campaign.csh
> It's still not entirely clear to me the minimum requirements for using globus.
> A specific library version must be passed to ncar_pylib on compute nodes.
On login nodes that is not needed.
> Extended globus endpoint lifetime to 30 days.
compress.csh
> Made the action (gzip or gunzip) an argument instead of hard coded.
Required changes to assimilate.csh and repack_st_archive.csh.
> Generalized to work under PBS (mpiexec_mpt) and slurm (mpirun)
repack_st_archive.csh
> Extensive upgrades and rewriting.
> It should only be run on casper, to have submit globus commands from the compute nodes.
> Listed numbers of tasks needed for various subsets of do_{data_sets}.
> Added section to archive {lnd,atm,ice,rof}/hist.
> Improved instructions.
> More setup error checks.
> Implemented a cmds_template, since the same archiving commands are used
in several places.
> Adapted to ncrcat's current bugs and fixed my mistaken use of -A.
> Changed variable name mpt_status to mpi_status to handle PBS and slurm.
> Focused the script on archiving a chosen year and month,
instead of assuming that all the files in archive are the right dates.
> Moved some removals of files and directories into the new purge.csh script,
which has to exist anyway to clean up after the asynchronous globus commands
have finished.
> Adapted to save_rest_freq being either a number or a day of the week.
> Refined the cmd_file commands to handle more error conditions more effectively.
> Fixed args to mv_to_campaign.csh, which were putting files in the wrong directories.
> Commented out the running of obs_diag and matlab scirpts,
since the plan is for this first repack_st_archive.csh to be
a less comprehensive, easier+safer to implement script (snort)
than the one I hope to use in the future.
But it still repackages and archives the obs_space files.
> Replaced echos of calls to mv_to_campaign.csh (for later use)
with actual calls (from casper's batch nodes).
> Added a section to archive the da.log files. (Other component's logs could be added).
purge.csh
> A script to remove files that are no longer needed after repack_st_archive.csh
and globus have finished archiving.
DART_config.template
> Added archiving scripts to the list to be copied to $CASEROOT.
assimilate.csh.template
> Fixed test of numerical save_rest_freq to find all instead of just the first.
> Improved handling of h0 and rh[0-9] files, prompted by history output from CLM.
> Less frequent ls of h0 files, to prevent poor performance when the directory
has thousands of files.
> Commented out the run_shadow section for the same reason.
> Removed the compression of obs_seq files because they are not huge+numerous
and need to be processed at the end of each month. Then they can be compressed
by repack_st_archive.csh.
Modified: DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/DART_config.template
===================================================================
--- DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/DART_config.template 2019-07-09 20:14:22 UTC (rev 13190)
+++ DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/DART_config.template 2019-07-09 22:15:00 UTC (rev 13191)
@@ -190,12 +190,15 @@
chmod 755 no_assimilate.csh
# chmod 755 perfect_model.csh
-if (-f ${DART_SCRIPTS_DIR}/compress.csh) then
- $COPY -f ${VERBOSE} ${DART_SCRIPTS_DIR}/compress.csh . || exit 43
-else
- echo "ERROR: no compress.csh in ${DART_SCRIPTS_DIR}"
- exit 45
-endif
+foreach script (compress.csh mv_to_campaign.csh repack_st_arch.csh \
+ purge.csh launch_cf.sh diags_batch.csh)
+ if (-f ${DART_SCRIPTS_DIR}/${script}) then
+ $COPY -f ${VERBOSE} ${DART_SCRIPTS_DIR}/${script} . || exit 43
+ else
+ echo "ERROR: no ${script} in ${DART_SCRIPTS_DIR}"
+ exit 45
+ endif
+end
# ==============================================================================
# Stage the DART executables in the CESM execution root directory: EXEROOT
Modified: DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/assimilate.csh.template
===================================================================
--- DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/assimilate.csh.template 2019-07-09 20:14:22 UTC (rev 13190)
+++ DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/assimilate.csh.template 2019-07-09 22:15:00 UTC (rev 13191)
@@ -371,7 +371,7 @@
# Numbers can be tested inside the 'if' statement.
else if (`echo $save_rest_freq | grep '[0-9]'`) then
- if ($day_o_month == $save_rest_freq) set purge = 'false'
+ if (${day_o_month} % ${save_rest_freq} == 0) set purge = 'false'
endif
@@ -397,8 +397,16 @@
# Note that *cpl.h[ar]* are retained, and any h#, #>0.
echo "Removing unneeded restart file set (DD_SSSSS ${day_time}) from RUNDIR: "
- echo " ${CASE}"'*.{r,rs,rs1,rh0,h0}.*'"${day_time}"
- ${REMOVE} ${CASE}*.{r,rs,rs1,rh0,h0}.*${day_time}* &
+ # It's OK to remove rh# (all components) because only the last date is needed,
+ # while this is the 3rd to last date. Also, if the history output is averaged,
+ # the averaging period is (should be) the length of the forecast,
+ # so there will be no running averages stored in the rh# files.
+ # But the h0 files need to be kept if any user specified fields are in them.
+ # They will be handled by repack_st_archive.csh and/or purge.csh later.
+ echo " ${CASE}"'*.{r,rs,rs1,rh*}.*'"${day_time}"
+ ${REMOVE} ${CASE}*.{r,rs,rs1,rh[0-9]}.*${day_time}* &
+ # echo " ${CASE}"'*.{r,rs,rs1,rh0,h0}.*'"${day_time}"
+ # ${REMOVE} ${CASE}*.{r,rs,rs1,rh0,h0}.*${day_time}* &
# Handle .i. separately to avoid sweeping up .i.${scomp}_{in,out}put_{mean,sd,...} files.
echo " ${CASE}"'*.i.[0-9]*'"${day_time}"
@@ -431,7 +439,7 @@
# List the large files first (more efficient and reliable).
# There is another call farther down to compress the DART files every cycle.
echo "compress.csh started at `date`"
- ${CASEROOT}/compress.csh $CASE ${rm_date} $ensemble_size "clm2 cpl cam cice" "$stages_all"
+ ${CASEROOT}/compress.csh gzip $CASE ${rm_date} $ensemble_size "clm2 cpl cam cice" "$stages_all"
if ($status != 0) then
echo "compress.csh failed at `date`"
exit 55
@@ -445,7 +453,7 @@
set save_root = ${archive}/rest/${rm_date}
if (! -d $save_root) then
mkdir -p $save_root
- (${MOVE} ${CASE}*.{r,rs,rs1,rh0,h0}.*${day_time}* $save_root || exit 60) &
+ (${MOVE} ${CASE}*.{r,rs,rs1,rh[0-9]}.*${day_time}* $save_root || exit 60) &
(${MOVE} ${CASE}*.i.[0-9]*${day_time}* $save_root || exit 61) &
(${COPY} *output*inf*${day_time}* $save_root || exit 62) &
(${MOVE} *0001*${rm_log[$rm_slot]}* $save_root || exit 63) &
@@ -532,13 +540,16 @@
${LINK} ${CASE}.cam_0001.i.${ATM_DATE_EXT}.nc ${CAMINPUT} || exit 90
# All of the .h0. files contain the same PHIS field, so we can link to any of them.
-
-set hists = `${LIST} ${CASE}.cam_0001.h0.*.nc`
set MYSTRING = `grep cam_phis_filename input.nml`
set MYSTRING = `echo $MYSTRING | sed -e "s#[=,']# #g"`
set CAM_PHIS = $MYSTRING[2]
-${REMOVE} ${CAM_PHIS}
-${LINK} $hists[1] ${CAM_PHIS} || exit 100
+# Avoid doing LISTs in rundir; only search for an h0 file is cam_phis.nc doesn't exist.
+if (! -f ${CAM_PHIS}) then
+ # -r puts the file with the latest date first in the list.
+ # It won't be purged during this cycle by the cleanup (above).
+ set hists = `${LIST} -r ${CASE}.cam_0001.h0.*.nc`
+ ${COPY} $hists ${CAM_PHIS} || exit 100
+endif
# Now, actually check the inflation settings
@@ -947,7 +958,12 @@
endif
More information about the Dart-dev
mailing list