[Dart-dev] DART/branches Revision: 13135

[dart at ucar.edu]

raeder at ucar.edu
2019-04-27 12:59:42 -0600 (Sat, 27 Apr 2019)
902
A script to repackage CAM6+DART output into forms that are 
useful and efficient for storage and future use.
2 sections have been developed and work in the testing context:
"forcing" files in the form of coupler history files,
and restart data sets, including inflation.

The restart section will call mv_to_campaign.csh, 
which submits a request to globus to transfer files,
but that has not been tested in this version.
Instead it writes out the mv_to_campaign.csh command
that would/could be used.

This script should be run as a batch job, 
since it uses the cmdfile mechanism.
If it calls mv_to_campaign.csh, it must be run on casper,
since the cheyenne compute nodes do not have access to globus.

2 more sections will be added; obs space diagnostics
(obs_diag and matlab scripts must be run first),
and DART diagnostic files (esp/hist/..., atm/hist/$inst.e.$stages).

It has not been reviewed.




Added: DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/repack_st_arch.csh
===================================================================
--- DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/repack_st_arch.csh	                        (rev 0)
+++ DART/branches/reanalysis/models/cam-fv/shell_scripts/cesm2_1/repack_st_arch.csh	2019-04-27 18:59:42 UTC (rev 13135)
@@ -0,0 +1,466 @@
+#!/bin/tcsh
+
+# DART software - Copyright UCAR. This open source software is provided
+# by UCAR, "as is", without charge, subject to all terms of use at
+# http://www.image.ucar.edu/DAReS/DART/DART_download
+
+# DART $Id$
+
+#==========================================================================
+
+# >>> Run st_archive and obs_diags.csh before running this script. <<<
+# >>> Edit compress.csh to make it decompress the (cpl/hist) files,
+#     in cases where they are compressed (gzipped).
+# >>> Log in to globus
+# >>> Run this script from the CESM CASEROOT directory. <<<
+
+# Script to package files found in $DOUT_S_ROOT
+# after $reanalysis_cesm/st_archive has sorted them,
+# and obs_diags.csh has generated basic obs space diagnostics.
+# The resulting files will be moved to 
+#   > a project space for further analysis and use
+#   > Campaign Storage for temporary archiving, 
+#     until we want to send them to HPSS (tape).
+# Both destinations take time.  They are actually copies.
+
+#-----------------------------------------
+# Submitting the job to casper (or other NCAR DAV clusters?) requires using slurm.
+
+# Important things to know about slurm:
+#
+# sinfo     information about the whole slurm system
+# squeue    information about running jobs
+# sbatch    submitting a job
+# scancel   killing a job
+#
+#==========================================================================
+#
+#SBATCH --job-name=repack_st_archive
+#SBATCH --ntasks=1 
+#SBATCH --time=01:00:00
+#SBATCH --mail-type=END
+#SBATCH --mail-type=FAIL
+#SBATCH --mail-user=raeder at ucar.edu
+#SBATCH --account=P86850054
+#SBATCH --partition=dav
+# 
+#-----------------------------------------
+#PBS  -N repack_st_archive
+#PBS  -A P86850054
+# #PBS  -q premium
+#PBS  -q share
+# Resources I want:
+#    select=#nodes
+#    ncpus=#CPUs/node
+#    mpiprocs=#MPI_tasks/node
+# Request enough processors for command file commands to handle the larger of 
+#    > ensemble size x 5 (forcing file types), 
+#    > (ensemble size + 1(mean_sd_log)) x # dates being saved (4-5 / month.  )
+# 81->405 #PBS  -l select=12:ncpus=34:mpiprocs=34
+#PBS  -l select=1:ncpus=15:mpiprocs=15
+#PBS  -l walltime=02:00:00
+# Send email after a(bort) or e(nd)
+#PBS  -m ae
+#PBS  -M raeder at ucar.edu
+# Send standard output and error to this file.
+# It's helpful to use the $CASE name here.
+#PBS  -o repack_st_archive.eo
+#PBS  -j oe 
+#--------------------------------------------
+
+if ($?PBS_O_WORKDIR) then
+   cd $PBS_O_WORKDIR
+else if ($?SLURM_SUBMIT_DIR) then
+   cd $SLURM_SUBMIT_DIR
+endif
+
+setenv MPI_SHEPHERD true
+setenv date 'date --rfc-3339=ns'
+
+echo "Preamble at "`date`
+
+if (! -f CaseStatus) then
+   echo "ERROR: this script must be run from the CESM CASEROOT directory"
+   exit 1
+endif
+
+set CASEROOT       = $cwd
+set CASE           = $CASEROOT:t
+set local_arch     = `./xmlquery DOUT_S_ROOT --value`
+set ensemble_size  = `./xmlquery NINST_ATM   --value`
+
+set line = `grep -m 1 save_every_Mth_day_restart ./setup*`
+set save_rest_freq = $line[3]
+
+
+if (! -d $local_arch) then