[Tgcmgroup] tgcm models on bluevista

[Ben Foster]

Dear tgcmgroup at ucar.edu,

This is a getting started guide to running TIEGCM and TIMEGCM models
on NCAR's new IBM clustered SMP system, bluevista.

Bluevista has 72 8-way POWER5 nodes. This machine is similiar to bluesky, 
except faster (bluesky has POWER4 processors). An important operational 
difference is that bluevista uses the LSF (Load Sharing Facility) batch 
system (like that on lightning) rather than the LoadLeveler, which is used 
on bluesky.

SCD's homepage for bluevista provides information re hardware, software,
documentation, queues, etc: http://www.cisl.ucar.edu/computers/bluevista

Your login, password, and default project account key should be the same 
as on bluesky. Once you are logged in, take the following steps to set up 
your environment for tgcm:

1) Check to make sure you are using the C-shell: the command "echo $SHELL" 
   should yield /usr/bin/csh. If not, you can give the command "csh"
   to start the C-shell. To change your default shell, ssh to blueview.ucar.edu
   using your bluevista logon name and follow the instructions. 

2) Set up your .cshrc file: Add the TGCMROOT and TGCMDATA environment
   variables to your ~/.cshrc file, before the "set path" statement:

   # tgcm root and data dirs:
    setenv TGCMROOT /fis/hao/tgcm
    setenv TGCMDATA /hao/tgcm/data

   Then add the this line to the path env var: 
     $TGCMROOT/bin $TGCMROOT/bin.`uname -s`
   See ~foster/.cshrc on bluevista as an example.

   You can then add your favorite aliases to the .cshrc file. If you have
   trouble with your .cshrc, example files are on bv in /usr/local/skel.
   Don't forget to "source ~/.cshrc" to make the changes take effect, however,
   recall that sourcing the .cshrc file will generally not *unset* env vars,
   so when in doubt, logout and back in again.

Once your set up above is ready, take the following steps to make a default run
of tiegcm1.8. Here, "bv>" is the bluevista command line prompt, and "mylogin"
is your login name on bluevista: 

1) Go to your /ptmp directory and make a subdirectory for tiegcm1.8:
    bv> cd /ptmp/mylogin
    bv> mkdir tiegcm1.8
    bv> cd tiegcm1.8

2) Copy the example csh job script (this can be submitted on bluevista or 
bluesky):
    bv> cp $TGCMROOT/tiegcm1.8/scripts/tiegcm-ibm.job .

3) Check and optionally edit the LSF #BSUB commands in the job script
   (if you are submitting to bluesky instead of bluevista, edit the 
    LoadLeveler "#@" commands).

4) Submit the script to the LSF batch system:

    bv> bsub < tiegcm-ibm.job

   (If you are submitting to bluesky, use "llsubmit tiegcm-ibm.job")

   Or you can submit a 4-task test job to the interactive share queue 
   (30-minute limit) with:
    bv> bsub -I -q share -n 4 tiegcm-ibm.job

5) Track the job with the command "bjobs". Once the job has started, 
   use "bpeek" to see watch script output. Jobs can be killed with the
   "bkill" command. See the man pages for these commands. These are the 
   same LSF commands we use on lightning.

Please see http://www.cisl.ucar.edu/docs/bluevista/lsf/index.html
for information about LSF and #BSUB commands.

To run timegcm1.2, you can substitute "tiegcm1.8" above with "timegcm1.2",
however the example job script to copy is: 
$TGCMROOT/timegcm1.2/scripts/timegcm-ibm.job

Please note that tiegcm-ibm.job (tiegcm1.8), and timegcm-ibm.job (timegcm1.2)
job scripts now contain two sections for specifying resources for batch jobs:
One section has LoadLeveler commands (starting with "#@") for use on bluesky,
the other section contains LSF commands (starting with #BSUB) for use on
bluevista. LoadLeveler commands are ignored by bv, and LSF commands are ignored
by bs.

These new scripts are not guaranteed to work with older model versions, because
the configure script for that version may not know about bluevista. If you must
run an older version on bluevista, please see me.

You can use the post-processor tgcmproc_f90 on bluevista, but not the IDL 
processor (IDL is not available on bluevista).

--Ben


-----------------------------------------------------------------------
Ben Foster		      	High Altitude Observatory (HAO)
foster at ucar.edu			phone: 303-497-1595  fax: 303-497-1589  
Nat. Center for Atmos. Res.     P.O. Box 3000 Boulder CO 80307 USA
-----------------------------------------------------------------------