[Dart-dev] DART/branches Revision: 12189

[dart at ucar.edu]

thoar at ucar.edu
2017-12-08 11:45:28 -0700 (Fri, 08 Dec 2017)
589

These scripts have support for slurm, pbs, and lsf.
They are simpler and cleaner than previous versions.
batch_job_resource_explanation.txt  is intended to provide
a quick-start description for each queueing system.

The get_ocean_time.csh script extracts the 'ocean_time'
value from a ROMS netCDF file and puts it in a DSTART format
that can be used to rename files, modify namelists etc.

stage_experiment.csh is REQUIRED to be run initially.
It copies the template files into place and replaces
some strings with user supplied values and then makes
the resulting scripts executable. 




Modified: DART/branches/rma_trunk/models/ROMS/shell_scripts/advance_ensemble.csh.template
===================================================================
--- DART/branches/rma_trunk/models/ROMS/shell_scripts/advance_ensemble.csh.template	2017-12-07 21:48:13 UTC (rev 12188)
+++ DART/branches/rma_trunk/models/ROMS/shell_scripts/advance_ensemble.csh.template	2017-12-08 18:45:28 UTC (rev 12189)
@@ -6,12 +6,29 @@
 #
 # DART $Id$
 #
+# This script submits the jobs to advance the ROMS ensemble members using
+# a JOB ARRAY. EACH ENSEMBLE MEMBER GETS EXECUTED ON ITS OWN NODE(s).
+# The run-time control file for ROMS gets updated with the next filename
+# and DSTART appropriate for the next forecast cycle.
+#
+# The prerequisite for this script is that advance_ensemble.csh has been run.
+# Things to note: many strings are intended to be replaced when this
+# template gets copied and ultimately submitted. Anything that starts
+# with 'My' is a string that gets replaced in the "normal" ROMs fashion.
+# The next few are not standard.
+#
+# Myens_size             gets replaced in the 'stage_experiment.csh' script
+# SXXXXX                 gets replaced if this is a dependent job (with SBATCH)
+# FILTERJOBID            gets replaced by the job ID that must finish first
+# EXPERIMENT_DIRECTORY   gets replaced in the 'stage_experiment.csh' script
+#
 #==========================================================================
-# this script submits the jobs to advance the ROMS ensemble members using
-# a job array. Each ensemble member gets executed on its own node.
-#
+# SLURM directives 			sbatch advance_ensemble.csh
+# 					squeue -u $USER
+# 					scancel <jobnumber>
+#SBATCH --ignore-pbs
 #SBATCH --array=1-Myens_size
-#SBATCH --ntasks=16 
+#SBATCH --ntasks=16
 #SBATCH --time=10:00
 #SBATCH --exclude=node04
 #SBATCH --job-name=roms_filter
@@ -18,47 +35,72 @@
 #SBATCH --output=roms_filter-%A-%a.out
 #SBATCH --error=roms_filter-%A-%a.err
 #
-#SXXXXX --dependency=afterok:FILTERJOBID
 #==========================================================================
+# PBS directives                	qsub test_batch.csh
+# 					qstat -u $USER
+# 					qdel <jobnumber>
+#PBS -J 1-Myens_size
+#PBS -l walltime=00:10:00
+#PBS -q economy
+#PBS -l select=1:ncpus=16:mpiprocs=16
+#PBS -A P86850054
+#PBS -N roms_filter
 #
-# Things to note: many strings are intended to be replaced when this
-# template gets copied and ultimately submitted. Anything that starts
-# with 'My' is a string that gets replaced in the "normal" ROMs fashion.
-# The next few are not standard. 
-#
-# Myens_size             gets replaced in the 'stage_experiment.csh' script
-# SXXXXX                 gets replaced if this is a dependent job (with SBATCH)
-# FILTERJOBID            gets replaced by the job ID that must finish first
-# EXPERIMENT_DIRECTORY   gets replaced in the 'stage_experiment.csh' script
+#==========================================================================
+# LSF directives 			bsub < advance_ensemble.csh
+# 					bjobs
+# 					bkill <jobnumber>
+#BSUB -J roms_filter[1-Myens_size]
+#BSUB -o roms_filter.%J.%I.log
+#BSUB -N -u ${USER}@ucar.edu
+#BSUB -q small
+#BSUB -n 1
+#BSUB -W 0:10
+#BSUB -P P86850054
 
-# machine-specific dereferencing
-if ($?LS_SUBCWD) then
-   set     JOBNAME = $LSB_JOBNAME
-   set       JOBID = $LSB_JOBID
-   set    NODELIST = $LSB_SUB_HOST
-   set   LAUNCHCMD = "mpirun.lsf"
-else if ($?SLURM_JOB_ID) then
+#==========================================================================
+# STEP 0: figure out the platform-specific nuances.
+#==========================================================================
+if ($?SLURM_JOB_ID) then
+
    set     JOBNAME = $SLURM_JOB_NAME
    set       JOBID = $SLURM_JOBID
+   set ARRAY_INDEX = $SLURM_ARRAY_TASK_ID
    set    NODELIST = $SLURM_NODELIST
    set   LAUNCHCMD = "mpirun -np $SLURM_NTASKS -bind-to core"
+
 else if ($?PBS_O_WORKDIR) then
-   set     JOBNAME = 
-   set       JOBID = 
-   set    NODELIST = 
-   set   LAUNCHCMD = "mpirun -np $SLURM_NTASKS -bind-to core"
+
+   env | grep PBS | sort
+