[Wrf-users] problem of running WRF with 2+ threads on Mac

[Craig Mattocks]

Hi Jian,

WRF runs fine in hybrid shared + distributed memory mode on my MacBook Pro laptop (Intel Core i7, Sandy Bridge, SSD) under Mac OS X Lion. I am getting about a 10% speed boost over pure MPI.

Make sure you follow the instructions in the 'configure.wrf' file:

# increase stack size; also note that for OpenMP, set environment OMP_STACKSIZE 4G or greater
LDFLAGS_LOCAL = -ip -Wl,-stack_addr,0xF10000000 -Wl,-stack_size,0x64000000

by setting:

setenv OMP_STACKSIZE 4G

before you run your simulation.

I am also using these settings in my .cshrc file:

###############################################
# Allow unlimited memory for big executables: #
###############################################
limit datasize unlimited
limit descriptors 8912
limit memorylocked unlimited
limit memoryuse unlimited
limit stacksize unlimited

mpich2 (http://www.mcs.anl.gov/research/projects/mpich2/) runs much faster than OpenMPI for me, but beware that you have to roll back to version 12.04 of the Intel Fortran compiler (instead of 12.1.x) to build the shared library version (I have reported this bug, but Intel warned me that it will not be fixed in the immediate future). Also, remove/rename the default Apple-installed OpenMPI compilers in your /usr/bin directory:

sudo \rm /usr/bin/opal_wrapper
sudo \rm /usr/bin/orte-ps
sudo \rm /usr/bin/orte-iof
sudo \rm /usr/bin/orte-clean
sudo \rm /usr/bin/orterun
sudo \rm /usr/bin/orted
sudo \rm /usr/bin/otfmerge
sudo \rm /usr/bin/otfinfo
sudo \rm /usr/bin/otfdump
sudo \rm /usr/bin/otfdecompress -> otfcompress
sudo \rm /usr/bin/otfconfig
sudo \rm /usr/bin/otfcompress
sudo \rm /usr/bin/otfaux
sudo \rm /usr/bin/vtunify
sudo \rm /usr/bin/vtfilter
sudo \rm /usr/bin/vtf90
sudo \rm /usr/bin/vtf77
sudo \rm /usr/bin/vtcxx
sudo \rm /usr/bin/vtcc
sudo \rm /usr/bin/opari
sudo \rm /usr/bin/ompi_info
sudo \rm /usr/bin/mpif90-vt
sudo \rm /usr/bin/mpif90
sudo \rm /usr/bin/mpif77-vt
sudo \rm /usr/bin/mpif77
sudo \rm /usr/bin/mpicxx-vt
sudo \rm /usr/bin/mpicxx
sudo \rm /usr/bin/mpicc-vt
sudo \rm /usr/bin/mpicc
sudo \rm /usr/bin/mpic++-vt
sudo \rm /usr/bin/mpic++
sudo \rm /usr/bin/mpiCC-vt
sudo \rm /usr/bin/mpiCC
sudo \rm /usr/bin/ompi-server
sudo \rm /usr/bin/ompi-ps
sudo \rm /usr/bin/ompi-iof
sudo \rm /usr/bin/ompi-clean
sudo \rm /usr/bin/mpirun
sudo \rm /usr/bin/mpiexec

Some good input data for testing:

wget http://www.mmm.ucar.edu/wrf_tmp/friendly/VENDOR_small_arw_ic.tar

Hope this helps,
Craig


On Friday, 18 November 2011 at 13:51:11 -0600 Jian Sun <jians at umn.edu> wrote:

> Hi everyone,
> 
> I have a problem of running WRF on my Mac pro with Lion (Xcode 4.2). I am
> using intel fortran composer XE and GCC (4.2).
> 
> I am able to run WRF in serial mode or in smpar mode with single thread.
> But segmentation fault occurs (forrtl: severe (174): SIGSEGV, segmentation
> fault occurred) when i try to run it with 2 or more threads. After a lot
> searching, the most possible reason seems to be about the stack size. I did
> "unlimit" in the command line, which is supposed to free the stack size,
> but there is hard upper limit in mac (64MB). Then I adjusted LDFLAGS_LOCAL
> option in configure.wrf, increasing -stack_size to 0x1F0000000 (about 8GB). I
> still cannot make it run with 2+ threads and get the same error.
> 
> Any help or advice is truly appreciated
> 
> Jian Sun