[Tgcmgroup] Recommendations for bluevista

[Ben Foster]

Hi tgcmgroup:

I have two recommendations for all TIEGCM/TIMEGCM runs on bluevista.
These do not apply to bluesky, but will apply to blueice, when it
becomes available.

I have added these recommendations to the default .job files for two 
versions of tiegcm and timegcm at both HAO and SCD, so you can refer
to them, or copy and modify as needed for your jobs:

$TGCMROOT/tiegcm1.8/scripts/tiegcm-ibm.job 
$TGCMROOT/timegcm1.2/scripts/timegcm-ibm.job 

Thanks to Siddhartha Ghosh for making these suggestions and assisting in
testing and ongoing performance evaluation.

#-----------------------------------------------------------------------

The following environment variable settings are recommended on
all TGCM models run on bluevista:

#
# 11/29/06: add env vars for bluevista (suggested by Siddhartha)
#
if ($nodename =~ bv*) then # bluevista
  setenv MP_STDINMODE 0
  setenv MP_SHARED_MEMORY yes
  setenv MEMORY_AFFINITY MCM
  setenv AIXTHREAD_SCOPE S
  setenv MALLOCMULTIHEAP true
endif

This can be added in your .job script, before the #BSUB commands
(just after the similiar set of env vars for bluesky).

When the new blueice system is available (and bluesky is retired), 
we can remove the conditional on the nodename.

#-----------------------------------------------------------------------

Simultaneous Multi-Threading (SMT), available on the IBM POWER5 architecture,
adds a virtual processor to each physical processor, doubling the number 
of active threads on a node. So when using SMT you can request 16 processors 
per node rather than the physical 8, using the #BSUB -R option.

See also: http://www.cisl.ucar.edu/docs/bluevista/run.html#smt_use

To take advantage of SMT on bluevista, try one of the following 2 
examples (in your .job scripts):

#
#BSUB -n 48                 # 48 procs
#BSUB -R "span[ptile=16]"   # 3 nodes w/ SMT (Symmetric Multi-Threading)
#
##BSUB -n 32                # 32 procs
##BSUB -R "span[ptile=16]"  # 2 nodes w/ SMT
#

The 2.5 degree timegcm model scales to 48 processors, as set above, but 
I have not tested the other models, so please experiment with these and 
other combinations.

#-----------------------------------------------------------------------

Most of the performance improvement will come from setting the env vars
rather than SMT.

Although the above applies only to bluevista, a job script with the above 
changes should continue to run on bluesky, because bluesky recognizes only 
LoadLeveler #@ directives, ignoring the LSF #BSUB directives.

When bluesky is retired early next year, we can remove all #@ directives,
since LoadLeveler is being retired, and all scd systems will run LSF.



-----------------------------------------------------------------------
Ben Foster		      	High Altitude Observatory (HAO)
foster at ucar.edu			phone: 303-497-1595  fax: 303-497-1589  
Nat. Center for Atmos. Res.     P.O. Box 3000 Boulder CO 80307 USA
-----------------------------------------------------------------------