[Dart-dev] DART/branches Revision: 12957

[dart at ucar.edu]

raeder at ucar.edu
2019-01-12 17:35:46 -0700 (Sat, 12 Jan 2019)
526
Merged results of a code review (Tim, Nancy, Kevin) with more tidying and documentation.
Tested for cpl history files, but not others yet.
RUNCMD had to be divided to allow the -n option to mpiexec_mpt.
This prevents null commands from being sent to extra tasks,
since the default of mpiexec_mpt is to use all available tasks,
regardless of mycmdfile.
The 'here' document syntax using a quoted EndOfFile doesn't seem 
to work on cheyenne or my laptop.  Variations also failed.
I settled on unquoted EndOfFile and lots of \s.




Modified: DART/branches/recam/models/cam-fv/shell_scripts/cesm2_1/compress.csh
===================================================================
--- DART/branches/recam/models/cam-fv/shell_scripts/cesm2_1/compress.csh	2019-01-10 23:18:35 UTC (rev 12956)
+++ DART/branches/recam/models/cam-fv/shell_scripts/cesm2_1/compress.csh	2019-01-13 00:35:46 UTC (rev 12957)
@@ -11,31 +11,47 @@
 #PBS  -q premium
 #PBS  -l select=16:ncpus=36:mpiprocs=36
 #PBS  -l walltime=00:20:00
-#PBS  -o CESM2_1_80_3node.out
+#PBS  -o compress_2010-07-17-21600.out
 #PBS  -j oe 
 
-# Most often called by assimilate.csh to compress a batch of files. 
-# Compression will depend on the file type,
-# and may include lossy compression.
-# > Execute something like this 2 places:
-#      Before archiving a restart set to archive/rest; all large restart files.
-#         clm2.r. compress 93.5% of 7.7 Gb in 1:23.
-#         .i. files only gzip compress ~12% in 24 seconds
-#      Every cycle: 
-#         all the cpl history (forcing) files.
-#       ? DART output
-#            stages of state files  (worth it?  only 15% in 10s)
-#               mean, sd  (no inst)
-#            obs_seq.final (no inst)   70% of 1 Gb in 35 sec
-#            inflations (no inst)  (should be like stages, but not many files)
-#      These 2 have different dates! $save_date  vs. $ATM_DATE_EXT
-# 
-# ? In which directory will assimilate.csh be when this executes?
-# ? Where will it find compres.csh?
+# -------------------------
+# Purpose
 
+# (De)Compress files from the forecast or assimilation. 
+
+# -------------------------
+# Method
+
+# This script can compress as well as decompress sets of files.
+# When called from a script (normally assimilate.csh), it compresses the files.
+# When called from the command line, it can uncompress sets of files,
+# if this script has been edited to use the right metadata to
+# construct the expected data directories.
+
+# It operates on all files in parallel using the "command file" mechanism.
+# The file contains a separate task on each line.
+# It has syntax appropriate for the launch script (below), 
+# which may not be the same as this script.  This must include
+# syntax to put stderr and stdout in a single file, e.g. ( &> ) for bash.
+# It is dispatched by the local MPI script to perform N simultaneous operations. 
+
+# Compression method can depend on the file type, and may include lossy compression.
+# It is most often called by assimilate.csh, but can be run as a batch job.
+# Assimilate.csh runs this in 2 places:
+# 1) Every cycle: 
+#    +  all the cpl history (forcing) files.
+#    +  DART output
+#       >  stages of state files  
+#             mean, sd  (no inst)
+#       >  obs_seq.final (no inst)   70% of 1 Gb in 35 sec
+#       >  not inflations (no inst)  
+# 1) Before archiving a restart set to archive/rest; all large restart files.
+
+# -------------------------
+
 if ($#argv == 5) then
    # Called from assimilate.csh (or other script).
-   set comp_cmd = 'gzip -k'
+   set comp_cmd      = 'gzip'
    set case_name     = $1
    set ymds          = $2
    set ensemble_size = $3
@@ -45,14 +61,15 @@
 
 else if ($#argv == 0) then
    # Edit these and run as a batch job.
-   set comp_cmd      = 'gunzip -k'
+   set comp_cmd      = 'gunzip'
    set case_name     = CESM2_1_80_3node
    set ymds          = 2010-07-17-64800
    set ensemble_size = 80
-   set sets          = (clm2 cpl cam cice)
+   set sets          = (hist dart)
+   # set sets          = (clm2 cpl cam cice)
    set stages        = (preassim output)
-   set data_dir      = /glade/scratch/${USER}/${case_name}/run/Compressed2
-
+   set data_dir      = /gpfs/fs1/scratch/raeder/${case_name}/run
+   # set data_dir      = /gpfs/fs1/scratch/raeder/${case_name}/archive/rest/2010-07-15-00000
 else 
    echo "Usage: Cannot run interactively."
    echo "       > vi compress.csh {set it up}; qsub compress.csh"
@@ -63,16 +80,6 @@
    exit 17
 endif
 
-set cmd = `echo $comp_cmd | cut -d' ' -f1`
-if ($cmd == 'gzip') then
-   set ext = ''
-else if ($cmd == 'gunzip') then
-   set ext = '.gz'